{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "# An Overview of Quantized Activations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this second tutorial, we take a deeper look at quantized activation.  \n",
    "We were already introduced to quantized activations in the previous tutorial, when we looked at input and output quantization of `QuantConv2d` with the `Int8ActPerTensorFloat` quantizer. The same result can be obtained with different syntax by coupling `QuantConv2d` with `QuantIdentity` layers, which by default uses the `Int8ActPerTensorFloat` quantizer.\n",
    "As an example, we compare - on the *same input* - the result of `QuantConv2d` with `output_quant` enabled with the result of a `QuantConv2d` followed by a `QuantIdentity`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:05.747069Z",
     "iopub.status.busy": "2025-05-09T14:48:05.746100Z",
     "iopub.status.idle": "2025-05-09T14:48:05.770581Z",
     "shell.execute_reply": "2025-05-09T14:48:05.768393Z"
    }
   },
   "outputs": [],
   "source": [
    "# helpers\n",
    "def assert_with_message(condition):\n",
    "    assert condition\n",
    "    print(condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:05.779449Z",
     "iopub.status.busy": "2025-05-09T14:48:05.778709Z",
     "iopub.status.idle": "2025-05-09T14:48:13.271388Z",
     "shell.execute_reply": "2025-05-09T14:48:13.268279Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/torch/_tensor.py:1255: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525541990/work/c10/core/TensorImpl.h:1758.)\n",
      "  return super(Tensor, self).rename(names)\n",
      "[W NNPACK.cpp:53] Could not initialize NNPACK! Reason: Unsupported hardware.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from brevitas.nn import QuantConv2d, QuantIdentity\n",
    "from brevitas.quant.scaled_int import Int8ActPerTensorFloat\n",
    "\n",
    "torch.manual_seed(0)  # set a seed to make sure the random weight init is reproducible\n",
    "output_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3), output_quant=Int8ActPerTensorFloat)\n",
    "\n",
    "torch.manual_seed(0)  # reproduce the same random weight init as above\n",
    "default_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3))\n",
    "output_identity_quant = QuantIdentity()\n",
    "\n",
    "inp = torch.randn(1, 2, 5, 5)\n",
    "out_tensor1 = output_quant_conv(inp)\n",
    "out_tensor2 = output_identity_quant(default_quant_conv(inp))\n",
    "\n",
    "assert_with_message(out_tensor1.isclose(out_tensor2).all().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe a similar behaviour if we enable input quantization too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.479178Z",
     "iopub.status.busy": "2025-05-09T14:48:13.478156Z",
     "iopub.status.idle": "2025-05-09T14:48:13.578257Z",
     "shell.execute_reply": "2025-05-09T14:48:13.575683Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(0)\n",
    "input_output_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3),\n",
    "    input_quant=Int8ActPerTensorFloat, output_quant=Int8ActPerTensorFloat)\n",
    "\n",
    "torch.manual_seed(0)\n",
    "default_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3))\n",
    "input_identity_quant = QuantIdentity()\n",
    "output_identity_quant = QuantIdentity()\n",
    "\n",
    "inp = torch.randn(1, 2, 5, 5)\n",
    "out_tensor1 = input_output_quant_conv(inp)\n",
    "out_tensor2 = output_identity_quant(default_quant_conv(input_identity_quant(inp)))\n",
    "\n",
    "assert_with_message(out_tensor1.isclose(out_tensor2).all().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From an algorithmic point of view then the two different implementation are doing the same thing. However, as it will become clearer in later tutorials, there are currently some scenarios where picking one style over the other can make a difference when it comes to exporting to a format such as standard ONNX. In the meantime, we can just keep in mind that both alternatives exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it was the case with `QuantConv2d`, when we disable quantization of an activation, the layer behaves as its floating-point variant. In the case of `QuantIdentity`, that means behaving like an identity function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.587965Z",
     "iopub.status.busy": "2025-05-09T14:48:13.587273Z",
     "iopub.status.idle": "2025-05-09T14:48:13.603941Z",
     "shell.execute_reply": "2025-05-09T14:48:13.601330Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "disabled_quant_identity = QuantIdentity(act_quant=None)\n",
    "assert_with_message((inp == disabled_quant_identity(inp)).all().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, as it was the case for `QuantConv2d`, quantized activation layers can also return a `QuantTensor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.613366Z",
     "iopub.status.busy": "2025-05-09T14:48:13.612300Z",
     "iopub.status.idle": "2025-05-09T14:48:13.658661Z",
     "shell.execute_reply": "2025-05-09T14:48:13.656117Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntQuantTensor(value=tensor([[[[-0.4566, -0.5707, -0.5517,  0.5897,  1.5409],\n",
       "          [ 0.5136, -0.5897, -0.5707,  0.1902, -0.0761],\n",
       "          [-0.4946, -1.5029, -0.1902,  0.4376,  1.3317],\n",
       "          [-1.6361,  2.0736,  1.7122,  2.3780, -1.1224],\n",
       "          [-0.3234, -1.0844, -0.0761, -0.0951, -0.7610]],\n",
       "\n",
       "         [[-1.5980,  0.0190, -0.7419,  0.1902,  0.6278],\n",
       "          [ 0.6468, -0.2473, -0.5327,  1.1605,  0.4376],\n",
       "          [-0.7990, -1.2936, -0.7419, -1.3127, -0.2283],\n",
       "          [-2.4351, -0.0761,  0.2283,  0.7990, -0.1902],\n",
       "          [-0.3615, -1.2175, -0.6278, -0.4566,  1.9214]]]],\n",
       "       grad_fn=<MulBackward0>), scale=tensor(0.0190, grad_fn=<AbsBinarySignGradFnBackward>), zero_point=tensor(0.), bit_width=tensor(8.), signed_t=tensor(True), training_t=tensor(True))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "return_quant_identity = QuantIdentity(return_quant_tensor=True)\n",
    "out_tensor = return_quant_identity(inp)\n",
    "out_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.667098Z",
     "iopub.status.busy": "2025-05-09T14:48:13.666229Z",
     "iopub.status.idle": "2025-05-09T14:48:13.679818Z",
     "shell.execute_reply": "2025-05-09T14:48:13.677189Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "assert_with_message(out_tensor.is_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, a `QuantIdentity` with quantization disabled behaves like an identity function also when a `QuantTensor` is passed in. However, depending on whather `return_quant_tensor` is set to `False` or not, quantization metadata might be stripped out, i.e. the input `QuantTensor` is going to be returned as an implicitly quantized `torch.Tensor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.690060Z",
     "iopub.status.busy": "2025-05-09T14:48:13.689087Z",
     "iopub.status.idle": "2025-05-09T14:48:13.709120Z",
     "shell.execute_reply": "2025-05-09T14:48:13.706364Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[-0.4566, -0.5707, -0.5517,  0.5897,  1.5409],\n",
       "          [ 0.5136, -0.5897, -0.5707,  0.1902, -0.0761],\n",
       "          [-0.4946, -1.5029, -0.1902,  0.4376,  1.3317],\n",
       "          [-1.6361,  2.0736,  1.7122,  2.3780, -1.1224],\n",
       "          [-0.3234, -1.0844, -0.0761, -0.0951, -0.7610]],\n",
       "\n",
       "         [[-1.5980,  0.0190, -0.7419,  0.1902,  0.6278],\n",
       "          [ 0.6468, -0.2473, -0.5327,  1.1605,  0.4376],\n",
       "          [-0.7990, -1.2936, -0.7419, -1.3127, -0.2283],\n",
       "          [-2.4351, -0.0761,  0.2283,  0.7990, -0.1902],\n",
       "          [-0.3615, -1.2175, -0.6278, -0.4566,  1.9214]]]],\n",
       "       grad_fn=<AliasBackward0>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_torch_tensor = disabled_quant_identity(out_tensor)\n",
    "out_torch_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.718864Z",
     "iopub.status.busy": "2025-05-09T14:48:13.718050Z",
     "iopub.status.idle": "2025-05-09T14:48:13.746354Z",
     "shell.execute_reply": "2025-05-09T14:48:13.743515Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntQuantTensor(value=tensor([[[[-0.4566, -0.5707, -0.5517,  0.5897,  1.5409],\n",
       "          [ 0.5136, -0.5897, -0.5707,  0.1902, -0.0761],\n",
       "          [-0.4946, -1.5029, -0.1902,  0.4376,  1.3317],\n",
       "          [-1.6361,  2.0736,  1.7122,  2.3780, -1.1224],\n",
       "          [-0.3234, -1.0844, -0.0761, -0.0951, -0.7610]],\n",
       "\n",
       "         [[-1.5980,  0.0190, -0.7419,  0.1902,  0.6278],\n",
       "          [ 0.6468, -0.2473, -0.5327,  1.1605,  0.4376],\n",
       "          [-0.7990, -1.2936, -0.7419, -1.3127, -0.2283],\n",
       "          [-2.4351, -0.0761,  0.2283,  0.7990, -0.1902],\n",
       "          [-0.3615, -1.2175, -0.6278, -0.4566,  1.9214]]]],\n",
       "       grad_fn=<AliasBackward0>), scale=tensor(0.0190, grad_fn=<AbsBinarySignGradFnBackward>), zero_point=tensor(0.), bit_width=tensor(8.), signed_t=tensor(True), training_t=tensor(True))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "return_disabled_quant_identity = QuantIdentity(act_quant=None, return_quant_tensor=True)\n",
    "identity_out_tensor = return_disabled_quant_identity(out_tensor)\n",
    "identity_out_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on from `QuantIdentity`, let's take a look at `QuantReLU`. Anything we said so far about `QuantIdentity` also applies to `QuantReLU`. The difference though is that `QuantReLU` implements a ReLU function followed by quantization, while `QuantIdentity` is really just the quantization operator. Additionally, by default `QuantReLU` adopts the `Uint8ActPerTensorFloat`, meaning that the output of quantization is *unsigned*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.759866Z",
     "iopub.status.busy": "2025-05-09T14:48:13.758213Z",
     "iopub.status.idle": "2025-05-09T14:48:13.809212Z",
     "shell.execute_reply": "2025-05-09T14:48:13.806707Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntQuantTensor(value=tensor([[[[0.0000, 0.0000, 0.0000, 0.5974, 1.5402],\n",
       "          [0.5041, 0.0000, 0.0000, 0.1867, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.4481, 1.3255],\n",
       "          [0.0000, 2.0817, 1.7083, 2.3804, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000]],\n",
       "\n",
       "         [[0.0000, 0.0187, 0.0000, 0.1867, 0.6254],\n",
       "          [0.6348, 0.0000, 0.0000, 1.1668, 0.4387],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.2334, 0.7935, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 1.9230]]]], grad_fn=<MulBackward0>), scale=tensor(0.0093, grad_fn=<AbsBinarySignGradFnBackward>), zero_point=tensor(0.), bit_width=tensor(8.), signed_t=tensor(False), training_t=tensor(True))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from brevitas.nn import QuantReLU\n",
    "\n",
    "return_quant_relu = QuantReLU(return_quant_tensor=True)\n",
    "return_quant_relu(inp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`QuantReLU`, like `QuantIdentity`, is also special compared to other non-linear quantized activation layers as it preserves the metadata of an input `QuantTensor` even when quantization is disabled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.820475Z",
     "iopub.status.busy": "2025-05-09T14:48:13.819594Z",
     "iopub.status.idle": "2025-05-09T14:48:13.842249Z",
     "shell.execute_reply": "2025-05-09T14:48:13.839883Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "tensor(True)\n",
      "tensor(True)\n",
      "tensor(True)\n"
     ]
    }
   ],
   "source": [
    "return_disabled_quant_relu = QuantReLU(act_quant=None, return_quant_tensor=True)\n",
    "relu_out_tensor = return_disabled_quant_relu(out_tensor)\n",
    "assert_with_message(relu_out_tensor.is_valid)\n",
    "assert_with_message(relu_out_tensor.scale == out_tensor.scale)\n",
    "assert_with_message(relu_out_tensor.zero_point == out_tensor.zero_point)\n",
    "assert_with_message(relu_out_tensor.bit_width == out_tensor.bit_width)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't apply to other layers like, say, `QuantSigmoid`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:13.853052Z",
     "iopub.status.busy": "2025-05-09T14:48:13.852179Z",
     "iopub.status.idle": "2025-05-09T14:48:16.479587Z",
     "shell.execute_reply": "2025-05-09T14:48:16.476663Z"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "QuantLayer is not correctly configured, check if warnings were raised",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[11], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mbrevitas\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mnn\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m QuantSigmoid\n\u001b[1;32m      3\u001b[0m return_disabled_quant_sigmoid \u001b[38;5;241m=\u001b[39m QuantSigmoid(act_quant\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, return_quant_tensor\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m----> 4\u001b[0m sigmoid_out_tensor \u001b[38;5;241m=\u001b[39m \u001b[43mreturn_disabled_quant_sigmoid\u001b[49m\u001b[43m(\u001b[49m\u001b[43mout_tensor\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m sigmoid_out_tensor\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/torch/nn/modules/module.py:1194\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1190\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1191\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1192\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1193\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1194\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1195\u001b[0m \u001b[38;5;66;03m# Do not call functions when jit is used\u001b[39;00m\n\u001b[1;32m   1196\u001b[0m full_backward_hooks, non_full_backward_hooks \u001b[38;5;241m=\u001b[39m [], []\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/quant_layer.py:53\u001b[0m, in \u001b[0;36mQuantNonLinearActLayer.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m out\n\u001b[1;32m     52\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mact_quant(quant_input)\n\u001b[0;32m---> 53\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpack_output\u001b[49m\u001b[43m(\u001b[49m\u001b[43mout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     54\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m out\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/mixin/base.py:101\u001b[0m, in \u001b[0;36mQuantLayerMixin.pack_output\u001b[0;34m(self, quant_output)\u001b[0m\n\u001b[1;32m     99\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpack_output\u001b[39m(\u001b[38;5;28mself\u001b[39m, quant_output: Union[Tensor, QuantTensor]) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Union[Tensor, QuantTensor]:\n\u001b[1;32m    100\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mreturn_quant_tensor:\n\u001b[0;32m--> 101\u001b[0m         \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(quant_output, QuantTensor), \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mQuantLayer is not correctly configured, check if warnings were raised\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m    102\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m quant_output\n\u001b[1;32m    103\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "\u001b[0;31mAssertionError\u001b[0m: QuantLayer is not correctly configured, check if warnings were raised"
     ]
    }
   ],
   "source": [
    "from brevitas.nn import QuantSigmoid\n",
    "\n",
    "return_disabled_quant_sigmoid = QuantSigmoid(act_quant=None, return_quant_tensor=True)\n",
    "sigmoid_out_tensor = return_disabled_quant_sigmoid(out_tensor)\n",
    "sigmoid_out_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something to always keep in mind is that the non-linearity of a quantized activation layer is always called on the *dequantized* representation of the input.\n",
    "For example, let's say we first quantize a floating-point `torch.Tensor` with an unsigned shifted quantizer such as `ShiftedUint8ActPerTensorFloat`, i.e. with zero-point such that the integer representation of its output is non-negative.\n",
    "Then, we pass this tensor as input to a `QuantReLU` with quantization *disabled*. The fact that the input to `QuantReLU` in its integer form is unsigned doesn't mean `QuantReLU` won't have any effect, as ReLU is called on the dequantized representation, which includes both *positive* and *negative* values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.491730Z",
     "iopub.status.busy": "2025-05-09T14:48:16.490846Z",
     "iopub.status.idle": "2025-05-09T14:48:16.546899Z",
     "shell.execute_reply": "2025-05-09T14:48:16.543677Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntQuantTensor(value=tensor([[[[0.0000, 0.0000, 0.0000, 0.5854, 1.5485],\n",
       "          [0.5099, 0.0000, 0.0000, 0.1888, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.4532, 1.3219],\n",
       "          [0.0000, 2.0772, 1.6996, 2.3794, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000]],\n",
       "\n",
       "         [[0.0000, 0.0189, 0.0000, 0.1888, 0.6232],\n",
       "          [0.6421, 0.0000, 0.0000, 1.1708, 0.4343],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "          [0.0000, 0.0000, 0.2266, 0.7931, 0.0000],\n",
       "          [0.0000, 0.0000, 0.0000, 0.0000, 1.9262]]]], grad_fn=<ReluBackward0>), scale=tensor(0.0189, grad_fn=<AbsBinarySignGradFnBackward>), zero_point=tensor(129., grad_fn=<WhereBackward0>), bit_width=tensor(8.), signed_t=tensor(False), training_t=tensor(True))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from brevitas.quant.shifted_scaled_int import ShiftedUint8ActPerTensorFloat\n",
    "\n",
    "shifted_quant_identity = QuantIdentity(act_quant=ShiftedUint8ActPerTensorFloat, return_quant_tensor=True)\n",
    "return_disabled_quant_relu = QuantReLU(act_quant=None, return_quant_tensor=True)\n",
    "return_disabled_quant_relu(shifted_quant_identity(inp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now consider the very common scenario of a `QuantConv2d` followed by a `ReLU` or `QuantReLU`.\n",
    "In particular, let's say we have a `QuantConv2d` with output quantization *enabled* followed by a `ReLU`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.557534Z",
     "iopub.status.busy": "2025-05-09T14:48:16.556570Z",
     "iopub.status.idle": "2025-05-09T14:48:16.614269Z",
     "shell.execute_reply": "2025-05-09T14:48:16.611157Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[0.0000, 0.0000, 0.0000],\n",
       "          [1.3134, 1.2557, 1.0392],\n",
       "          [0.4186, 0.0000, 0.0000]],\n",
       "\n",
       "         [[0.7361, 0.5340, 0.8516],\n",
       "          [0.2887, 0.3175, 0.0000],\n",
       "          [0.8949, 1.6743, 0.0722]],\n",
       "\n",
       "         [[0.0000, 0.0000, 0.0289],\n",
       "          [0.0000, 0.0000, 0.2021],\n",
       "          [0.0000, 0.0000, 0.4907]]]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(0)\n",
    "output_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3), output_quant=Int8ActPerTensorFloat)\n",
    "torch.relu(output_quant_conv(inp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compare it against a `QuantConv2d` with default settings (i.e. output quantization *disabled*), followed by a `QuantReLU` with default settings (i.e. activation quantization *enabled*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.624754Z",
     "iopub.status.busy": "2025-05-09T14:48:16.623899Z",
     "iopub.status.idle": "2025-05-09T14:48:16.675423Z",
     "shell.execute_reply": "2025-05-09T14:48:16.672520Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[0.0000, 0.0000, 0.0000],\n",
       "          [1.3078, 1.2555, 1.0397],\n",
       "          [0.4185, 0.0000, 0.0000]],\n",
       "\n",
       "         [[0.7454, 0.5427, 0.8566],\n",
       "          [0.2943, 0.3269, 0.0000],\n",
       "          [0.8893, 1.6674, 0.0785]],\n",
       "\n",
       "         [[0.0065, 0.0000, 0.0262],\n",
       "          [0.0000, 0.0000, 0.1962],\n",
       "          [0.0000, 0.0000, 0.4839]]]], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(0)\n",
    "default_quant_conv = QuantConv2d(\n",
    "    in_channels=2, out_channels=3, kernel_size=(3,3))\n",
    "default_quant_relu = QuantReLU()\n",
    "default_quant_relu(default_quant_conv(inp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the results are close but not quite the same.  \n",
    "In the first case, we quantized the output of `QuantConv2d` with an 8-bit signed quantizer, and then we passed it through a `ReLU`, meaning that half of the numerical range covered by the signed quantizer is now lost, and by all practical means the output can now be treated as a 7-bit unsigned number (although it's not explicitly marked as such).\n",
    "In the second case, we perform unsigned 8-bit quantization after `ReLU`. Because the range covered by the quantizer now includes only non-negative numbers, we don't waste a bit as in the previous case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regarding some premade activation quantizers, such as `Uint8ActPerTensorFloat`, `ShiftedUint8ActPerTensorFloat`, and `Int8ActPerTensorFloat`, a word of caution that anticipates some of the themes of the next tutorial.\n",
    "To minimize user interaction, Brevitas initializes scale and zero-point by collecting statistics for a number of training steps (by default 30). This can be seen as a sort of very basic calibration step, although it typically happens during training and with quantization already enabled. These statistics are accumulated in an exponential moving average that at end of the collection phase is used to initialize a learned *parameter*.\n",
    "During the collection phase then, the quantizer behaves differently between `train()` and `eval()` mode. In `train()` mode, the statistics for that particular batch are returned. In `eval()` mode, the exponential moving average is returned. After the collection phase is over the learned parameter is returned in both execution modes.\n",
    "We can easily observe this behaviour with an example. Let's first define a quantized activation and two random input tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.686929Z",
     "iopub.status.busy": "2025-05-09T14:48:16.685777Z",
     "iopub.status.idle": "2025-05-09T14:48:16.708202Z",
     "shell.execute_reply": "2025-05-09T14:48:16.705642Z"
    }
   },
   "outputs": [],
   "source": [
    "quant_identity = QuantIdentity(return_quant_tensor=True)\n",
    "inp1 = torch.randn(3, 3)\n",
    "inp2 = torch.randn(3, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then compare the output scale factor of the two tensors between `train()` and `eval()` mode. The ones in train mode in general are different. The ones in eval mode are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.720112Z",
     "iopub.status.busy": "2025-05-09T14:48:16.718902Z",
     "iopub.status.idle": "2025-05-09T14:48:16.743727Z",
     "shell.execute_reply": "2025-05-09T14:48:16.740391Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "out1_train = quant_identity(inp1)\n",
    "out2_train = quant_identity(inp2)\n",
    "assert_with_message(not out1_train.scale.isclose(out2_train.scale).item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.754370Z",
     "iopub.status.busy": "2025-05-09T14:48:16.752502Z",
     "iopub.status.idle": "2025-05-09T14:48:16.774367Z",
     "shell.execute_reply": "2025-05-09T14:48:16.772023Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "quant_identity.eval()\n",
    "out1_eval = quant_identity(inp1)\n",
    "out2_eval = quant_identity(inp2)\n",
    "assert_with_message(out1_eval.scale.isclose(out2_eval.scale).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the only layer that is an exception to this is `QuantHardTanh`. That is because the interface to `torch.nn.HardTanh` already requires users to manually specify `min_val` and `max_val`, so Brevitas preserves that both when quantization is enabled or disabled. With quantization enabled, by default those values are used for initialization, but then the range is learned. Let's look at an example. Run the cell below, and we expect it to throw an error because of missing attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:16.786909Z",
     "iopub.status.busy": "2025-05-09T14:48:16.785046Z",
     "iopub.status.idle": "2025-05-09T14:48:17.440933Z",
     "shell.execute_reply": "2025-05-09T14:48:17.437935Z"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "DependencyError",
     "evalue": "'Int8ActPerTensorFloatMinMaxInit' can not resolve attribute 'max_val' while building 'scaling_init_impl'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mDependencyError\u001b[0m                           Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[18], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mbrevitas\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mnn\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m QuantHardTanh\n\u001b[0;32m----> 3\u001b[0m \u001b[43mQuantHardTanh\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/quant_activation.py:96\u001b[0m, in \u001b[0;36mQuantHardTanh.__init__\u001b[0;34m(self, act_quant, input_quant, return_quant_tensor, **kwargs)\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\n\u001b[1;32m     91\u001b[0m         \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m     92\u001b[0m         act_quant: Optional[ActQuantType] \u001b[38;5;241m=\u001b[39m Int8ActPerTensorFloatMinMaxInit,\n\u001b[1;32m     93\u001b[0m         input_quant: Optional[ActQuantType] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m     94\u001b[0m         return_quant_tensor: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m     95\u001b[0m         \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m---> 96\u001b[0m     \u001b[43mQuantNLAL\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\n\u001b[1;32m     97\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m     98\u001b[0m \u001b[43m        \u001b[49m\u001b[43mact_impl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mHardtanh\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     99\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpassthrough_act\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    100\u001b[0m \u001b[43m        \u001b[49m\u001b[43minput_quant\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minput_quant\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    101\u001b[0m \u001b[43m        \u001b[49m\u001b[43mact_quant\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mact_quant\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    102\u001b[0m \u001b[43m        \u001b[49m\u001b[43mreturn_quant_tensor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_quant_tensor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    103\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/quant_layer.py:34\u001b[0m, in \u001b[0;36mQuantNonLinearActLayer.__init__\u001b[0;34m(self, act_impl, passthrough_act, input_quant, act_quant, return_quant_tensor, **kwargs)\u001b[0m\n\u001b[1;32m     32\u001b[0m QuantLayerMixin\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, return_quant_tensor)\n\u001b[1;32m     33\u001b[0m QuantInputMixin\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, input_quant, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m---> 34\u001b[0m \u001b[43mQuantNonLinearActMixin\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mact_impl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpassthrough_act\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mact_quant\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/mixin/act.py:66\u001b[0m, in \u001b[0;36mQuantNonLinearActMixin.__init__\u001b[0;34m(self, act_impl, passthrough_act, act_quant, act_proxy_prefix, act_kwargs_prefix, **kwargs)\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\n\u001b[1;32m     56\u001b[0m         \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m     57\u001b[0m         act_impl: Optional[Type[Module]],\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     61\u001b[0m         act_kwargs_prefix\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m     62\u001b[0m         \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m     63\u001b[0m     prefixed_kwargs \u001b[38;5;241m=\u001b[39m {\n\u001b[1;32m     64\u001b[0m         act_kwargs_prefix \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mact_impl\u001b[39m\u001b[38;5;124m'\u001b[39m: act_impl,\n\u001b[1;32m     65\u001b[0m         act_kwargs_prefix \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mpassthrough_act\u001b[39m\u001b[38;5;124m'\u001b[39m: passthrough_act}\n\u001b[0;32m---> 66\u001b[0m     \u001b[43mQuantProxyMixin\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\n\u001b[1;32m     67\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m     68\u001b[0m \u001b[43m        \u001b[49m\u001b[43mquant\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mact_quant\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     69\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxy_prefix\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mact_proxy_prefix\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     70\u001b[0m \u001b[43m        \u001b[49m\u001b[43mkwargs_prefix\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mact_kwargs_prefix\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     71\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxy_protocol\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mActQuantProxyProtocol\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     72\u001b[0m \u001b[43m        \u001b[49m\u001b[43mnone_quant_injector\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mNoneActQuant\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     73\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mprefixed_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     74\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/nn/mixin/base.py:52\u001b[0m, in \u001b[0;36mQuantProxyMixin.__init__\u001b[0;34m(self, quant, proxy_protocol, none_quant_injector, proxy_prefix, kwargs_prefix, **kwargs)\u001b[0m\n\u001b[1;32m     50\u001b[0m     quant_injector \u001b[38;5;241m=\u001b[39m quant\n\u001b[1;32m     51\u001b[0m     quant_injector \u001b[38;5;241m=\u001b[39m quant_injector\u001b[38;5;241m.\u001b[39mlet(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mfilter_kwargs(kwargs_prefix, kwargs))\n\u001b[0;32m---> 52\u001b[0m     quant \u001b[38;5;241m=\u001b[39m \u001b[43mquant_injector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mproxy_class\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquant_injector\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     53\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     54\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(quant, proxy_protocol):\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/proxy/runtime_quant.py:221\u001b[0m, in \u001b[0;36mActQuantProxyFromInjector.__init__\u001b[0;34m(self, quant_layer, quant_injector)\u001b[0m\n\u001b[1;32m    220\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, quant_layer, quant_injector):\n\u001b[0;32m--> 221\u001b[0m     \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mquant_layer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquant_injector\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    222\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcache_class \u001b[38;5;241m=\u001b[39m _CachedIO\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/proxy/runtime_quant.py:94\u001b[0m, in \u001b[0;36mActQuantProxyFromInjectorBase.__init__\u001b[0;34m(self, quant_layer, quant_injector)\u001b[0m\n\u001b[1;32m     93\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, quant_layer, quant_injector):\n\u001b[0;32m---> 94\u001b[0m     \u001b[43mQuantProxyFromInjector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquant_layer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquant_injector\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     95\u001b[0m     ActQuantProxyProtocol\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m)\n\u001b[1;32m     96\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mis_passthrough_act \u001b[38;5;241m=\u001b[39m _is_passthrough_act(quant_injector)\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/proxy/quant_proxy.py:80\u001b[0m, in \u001b[0;36mQuantProxyFromInjector.__init__\u001b[0;34m(self, quant_layer, quant_injector)\u001b[0m\n\u001b[1;32m     78\u001b[0m \u001b[38;5;66;03m# Use a normal list and not a ModuleList since this is a pointer to parent modules\u001b[39;00m\n\u001b[1;32m     79\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtracked_module_list \u001b[38;5;241m=\u001b[39m []\n\u001b[0;32m---> 80\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43madd_tracked_module\u001b[49m\u001b[43m(\u001b[49m\u001b[43mquant_layer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     81\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdisable_quant \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m     82\u001b[0m \u001b[38;5;66;03m# Torch.compile compatibility requires this\u001b[39;00m\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/proxy/quant_proxy.py:120\u001b[0m, in \u001b[0;36mQuantProxyFromInjector.add_tracked_module\u001b[0;34m(self, module)\u001b[0m\n\u001b[1;32m    118\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtracked_module_list\u001b[38;5;241m.\u001b[39mappend(module)\n\u001b[1;32m    119\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mupdate_tracked_modules()\n\u001b[0;32m--> 120\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minit_tensor_quant\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    121\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    122\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTrying to add None as a parent module.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/proxy/runtime_quant.py:149\u001b[0m, in \u001b[0;36mActQuantProxyFromInjectorBase.init_tensor_quant\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    148\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21minit_tensor_quant\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m--> 149\u001b[0m     tensor_quant \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mquant_injector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtensor_quant\u001b[49m\n\u001b[1;32m    150\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mact_impl\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mquant_injector:\n\u001b[1;32m    151\u001b[0m         act_impl \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mquant_injector\u001b[38;5;241m.\u001b[39mact_impl\n",
      "File \u001b[0;32m/proj/xlabs/users/nfraser/opt/miniforge3/envs/20231115_brv_pt1.13.1/lib/python3.10/site-packages/brevitas/inject/__init__.py:129\u001b[0m, in \u001b[0;36m_ExtendedInjectorType.__getattr__\u001b[0;34m(cls, attrname)\u001b[0m\n\u001b[1;32m    126\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    127\u001b[0m         message \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{!r}\u001b[39;00m\u001b[38;5;124m can not resolve attribute \u001b[39m\u001b[38;5;132;01m{!r}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mformat(\n\u001b[1;32m    128\u001b[0m             \u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, current_attr)\n\u001b[0;32m--> 129\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m DependencyError(message)\n\u001b[1;32m    131\u001b[0m marker, attribute, args, have_defaults \u001b[38;5;241m=\u001b[39m spec\n\u001b[1;32m    133\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mset\u001b[39m(args)\u001b[38;5;241m.\u001b[39missubset(cached):\n",
      "\u001b[0;31mDependencyError\u001b[0m: 'Int8ActPerTensorFloatMinMaxInit' can not resolve attribute 'max_val' while building 'scaling_init_impl'"
     ]
    }
   ],
   "source": [
    "from brevitas.nn import QuantHardTanh\n",
    "\n",
    "QuantHardTanh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, we get an error concering a missing `max_val` attribute. Let's try to pass it then, together with `min_val`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.451905Z",
     "iopub.status.busy": "2025-05-09T14:48:17.450987Z",
     "iopub.status.idle": "2025-05-09T14:48:17.473000Z",
     "shell.execute_reply": "2025-05-09T14:48:17.470513Z"
    }
   },
   "outputs": [],
   "source": [
    "quant_hard_tanh = QuantHardTanh(max_val=1.0, min_val=-1.0, return_quant_tensor=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The layer is now correctly initialized. We can see that the output scale factors are all the same between `train()` and `eval()` mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.482178Z",
     "iopub.status.busy": "2025-05-09T14:48:17.481444Z",
     "iopub.status.idle": "2025-05-09T14:48:17.500277Z",
     "shell.execute_reply": "2025-05-09T14:48:17.498003Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "out1_train = quant_hard_tanh(inp1)\n",
    "quant_hard_tanh.eval()\n",
    "out2_eval = quant_hard_tanh(inp2)\n",
    "assert_with_message(out1_train.scale.isclose(out2_eval.scale).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In all of the examples that have currently been looked at in this tutorial, we have used per-tensor quantization. I.e., the output tensor of the activation, if quantized, was always quantized on a per-tensor level, with a single scale and zero-point quantization parameter per output tensor. However, one can also do per-channel quantization, where each output channel of the tensor has its own quantization parameters. In the example below, we look at per-tensor quantization of an input tensor that has 3 channels and 256 elements in the height and width dimensions. We purposely mutate the 1st channel to have its dynamic range be 3 times larger than the other 2 channels. We then feed it through a `QuantReLU`, whose default behavior is to quantize at a per-tensor granularity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.509132Z",
     "iopub.status.busy": "2025-05-09T14:48:17.508420Z",
     "iopub.status.idle": "2025-05-09T14:48:17.564975Z",
     "shell.execute_reply": "2025-05-09T14:48:17.561963Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.9998, grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_channels = 3\n",
    "inp3 = torch.rand(1, out_channels, 256, 256)  # (B, C, H, W)\n",
    "inp3[:, 0, :, :] *= 3\n",
    "\n",
    "per_tensor_quant_relu = QuantReLU(return_quant_tensor=True)\n",
    "out_tensor = per_tensor_quant_relu(inp3)\n",
    "out_tensor.scale * ((2**8) -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the per-tensor scale parameter has calibrated itself to provide a full quantization range of 3, matching that of the channel with the largest dynamic range. \n",
    "\n",
    "We can take a look at the `QuantReLU` object, and in particular look at what the `scaling_impl` object is composed of. It is responsible for gathering statistics for determining the quantization parameters, and we can see that its `stats_input_view_shape_impl` attribute is set to be an instance of `OverTensorView`. This is defined [here](https://github.com/Xilinx/brevitas/blob/200456825f3b4b8db414f2b25b64311f82d3991a/src/brevitas/core/function_wrapper/shape.py#L78), and serves to flatten out the observed tensor into a 1D tensor and, in this case, use the `AbsPercentile` observer to calculate the quantization parameters during the gathering statistics stage of QAT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.577829Z",
     "iopub.status.busy": "2025-05-09T14:48:17.576792Z",
     "iopub.status.idle": "2025-05-09T14:48:17.595178Z",
     "shell.execute_reply": "2025-05-09T14:48:17.592386Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QuantReLU(\n",
       "  (input_quant): ActQuantProxyFromInjector(\n",
       "    (_zero_hw_sentinel): StatelessBuffer()\n",
       "  )\n",
       "  (act_quant): ActQuantProxyFromInjector(\n",
       "    (_zero_hw_sentinel): StatelessBuffer()\n",
       "    (fused_activation_quant_proxy): FusedActivationQuantProxy(\n",
       "      (activation_impl): ReLU()\n",
       "      (tensor_quant): RescalingIntQuant(\n",
       "        (int_quant): IntQuant(\n",
       "          (float_to_int_impl): RoundSte()\n",
       "          (tensor_clamp_impl): TensorClamp()\n",
       "          (delay_wrapper): DelayWrapper(\n",
       "            (delay_impl): _NoDelay()\n",
       "          )\n",
       "          (input_view_impl): Identity()\n",
       "        )\n",
       "        (scaling_impl): ParameterFromRuntimeStatsScaling(\n",
       "          (stats_input_view_shape_impl): OverTensorView()\n",
       "          (stats): _Stats(\n",
       "            (stats_impl): AbsPercentile()\n",
       "          )\n",
       "          (restrict_scaling): _RestrictValue(\n",
       "            (restrict_value_impl): FloatRestrictValue()\n",
       "          )\n",
       "          (restrict_threshold): _RestrictValue(\n",
       "            (restrict_value_impl): FloatRestrictValue()\n",
       "          )\n",
       "          (clamp_scaling): _ClampValue(\n",
       "            (clamp_min_ste): ScalarClampMinSte()\n",
       "          )\n",
       "          (restrict_inplace_preprocess): Identity()\n",
       "          (restrict_scaling_pre): Identity()\n",
       "          (restrict_threshold_pre): Identity()\n",
       "        )\n",
       "        (int_scaling_impl): IntScaling()\n",
       "        (zero_point_impl): ZeroZeroPoint(\n",
       "          (zero_point): StatelessBuffer()\n",
       "        )\n",
       "        (msb_clamp_bit_width_impl): BitWidthConst(\n",
       "          (bit_width): StatelessBuffer()\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_tensor_quant_relu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we initialise a new `QuantRelU` instance, but this time we specify that we desire per-channel quantization i.e. `scaling_per_output_channel=True`. This will implictly call `scaling_stats_input_view_shape_impl`, defined [here](https://github.com/Xilinx/brevitas/blob/200456825f3b4b8db414f2b25b64311f82d3991a/src/brevitas/quant/solver/common.py#L184), and will change the `QuantReLU` from using a per-tensor view when gathering stats to a per output channel view ([`OverOutputChannelView`](https://github.com/Xilinx/brevitas/blob/200456825f3b4b8db414f2b25b64311f82d3991a/src/brevitas/core/function_wrapper/shape.py#L52)). This simply permutes the tensor into a 2D tensor, with dim 0 equal to the number of output channels.\n",
    "\n",
    "To accomplish this, we also need to give it some extra information: `scaling_stats_permute_dims` and `per_channel_broadcastable_shape`. `scaling_stats_permute_dims` is responsible for defining how we do the permutation. `per_channel_broadcastable_shape` is necessary to understand along which dimensions the scale factor has to be broadcasted, so that the scale factor values are applied along the channel dimensions of the input.\n",
    "By default, PyTorch will broadcast along the first rightmost dimension for which the shapes of the two tensors match. To make sure that we apply the scale factor in our desired output channel dimension, we need to tell PyTorch how to correctly broadcast the scale factors. Therefore the scale factor will have as many dimensions as the input tensors, with all the shapes equal to 1 apart from the channel dimension.\n",
    "\n",
    "Below, we can see that in the per-channel ` QuantReLU` instance, the `stats_input_view_shape_impl` is now ` OverOutputChannelView`, and uses a `PermuteDims` [instance](https://github.com/Xilinx/brevitas/blob/200456825f3b4b8db414f2b25b64311f82d3991a/src/brevitas/core/function_wrapper/shape.py#L21) to do the permutation of the tensor to, in this case, a 2D tensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.608291Z",
     "iopub.status.busy": "2025-05-09T14:48:17.606595Z",
     "iopub.status.idle": "2025-05-09T14:48:17.639082Z",
     "shell.execute_reply": "2025-05-09T14:48:17.636252Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QuantReLU(\n",
       "  (input_quant): ActQuantProxyFromInjector(\n",
       "    (_zero_hw_sentinel): StatelessBuffer()\n",
       "  )\n",
       "  (act_quant): ActQuantProxyFromInjector(\n",
       "    (_zero_hw_sentinel): StatelessBuffer()\n",
       "    (fused_activation_quant_proxy): FusedActivationQuantProxy(\n",
       "      (activation_impl): ReLU()\n",
       "      (tensor_quant): RescalingIntQuant(\n",
       "        (int_quant): IntQuant(\n",
       "          (float_to_int_impl): RoundSte()\n",
       "          (tensor_clamp_impl): TensorClamp()\n",
       "          (delay_wrapper): DelayWrapper(\n",
       "            (delay_impl): _NoDelay()\n",
       "          )\n",
       "          (input_view_impl): Identity()\n",
       "        )\n",
       "        (scaling_impl): ParameterFromRuntimeStatsScaling(\n",
       "          (stats_input_view_shape_impl): OverOutputChannelView(\n",
       "            (permute_impl): PermuteDims()\n",
       "          )\n",
       "          (stats): _Stats(\n",
       "            (stats_impl): AbsPercentile()\n",
       "          )\n",
       "          (restrict_scaling): _RestrictValue(\n",
       "            (restrict_value_impl): FloatRestrictValue()\n",
       "          )\n",
       "          (restrict_threshold): _RestrictValue(\n",
       "            (restrict_value_impl): FloatRestrictValue()\n",
       "          )\n",
       "          (clamp_scaling): _ClampValue(\n",
       "            (clamp_min_ste): ScalarClampMinSte()\n",
       "          )\n",
       "          (restrict_inplace_preprocess): Identity()\n",
       "          (restrict_scaling_pre): Identity()\n",
       "          (restrict_threshold_pre): Identity()\n",
       "        )\n",
       "        (int_scaling_impl): IntScaling()\n",
       "        (zero_point_impl): ZeroZeroPoint(\n",
       "          (zero_point): StatelessBuffer()\n",
       "        )\n",
       "        (msb_clamp_bit_width_impl): BitWidthConst(\n",
       "          (bit_width): StatelessBuffer()\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_chan_quant_relu = QuantReLU(return_quant_tensor=True,\n",
    "                                scaling_per_output_channel=True,\n",
    "                                per_channel_broadcastable_shape=(1, out_channels, 1 , 1),\n",
    "                                scaling_stats_permute_dims=(1, 0, 2, 3),\n",
    "                                )\n",
    "per_chan_quant_relu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also observe the effect on the quantization parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-05-09T14:48:17.652901Z",
     "iopub.status.busy": "2025-05-09T14:48:17.651121Z",
     "iopub.status.idle": "2025-05-09T14:48:17.684061Z",
     "shell.execute_reply": "2025-05-09T14:48:17.681278Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[2.9998]],\n",
       "\n",
       "         [[1.0000]],\n",
       "\n",
       "         [[1.0000]]]], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_channel = per_chan_quant_relu(inp3)\n",
    "out_channel.scale * ((2**8) -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the number of elements in the quantization scale of the outputted tensor is now 3, matching those of the 3-channel tensor! Furthermore, we see that each channel has an 8-bit quantization range that matches its data distribution, which is much more ideal in terms of reducing quantization mismatch. However, it's important to note that some hardware providers don't efficiently support per-channel quantization in production, so it's best to check if your targetted hardware will allow per-channel quantization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, a reminder that mixing things up is perfectly legal and encouraged in Brevitas.\n",
    "For example, a `QuantIdentity` with `act_quant=Int8ActPerTensorFloatMinMaxInit` is equivalent to a default `QuantHardTanh`, or conversely a `QuantHardTanh` with `act_quant=Int8ActPerTensorFloat` is equivalent to a default `QuantIdentity`. This is allowed by the fact that - as it will be explained in the next tutorial - the same layer can accept different keyword arguments when different quantizers are set. So a QuantIdentity with `act_quant=Int8ActPerTensorFloatMinMaxInit` is going to expect arguments `min_val` and `max_val` the same way a default `QuantHardTanh` would."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
